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ABSTRACT 

Although multimedia technologies offer a great 
potential, educators and decision makers need to see evidence 
regarding the instructional effectiveness of multimedia in order to 
integrate it into the educational processes. The main purposes of 
this paper are: (1) to identify and discuss the main criticisms of 
media research, including the research format, characteristics, 
results, and recommendations made between the 1950s and early 1970s 
and between the mid-'1970s and the present, including the 
aptitude-treatment interaction approach; (2) to discuss the state of 
current interactive video simulation research in the light of 
criticisms being made; and (3) to identify the most important points 
that need to be considered by media researchers in order to avoid 
making similar mistakes in the future, particularly in the areas of 
cognitive and attitudinal effectiveness and the theoretical basis of 
the design of interactive video simulations. It is suggested that 
educators should abandon the task of trying to prove the 
effectiveness of multimedia simulations in comparison to other 
technologies and concentrate on exploring their potential in relation 
to learning tasks and learners' characteristics. (Contains 10 
references .) (ALF) 
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CO 1, INTRODUCTION 

As we all know, multimedia technologies, such as IV, CDI, and DVI, offer 
^ a great potential to educators. Interactive video (IV), for example, which 
is the most commonly available one, comprises quaJlty still and moving 
pictures, perfect digital sound, fast and accurate data processing power* 
and effective image and graphics display. Such a technical potential 
enables educators to create various types of IV simulations which, 
theoretically, are capable of meeting the requirements recommended by 
learning theories for effective learning processes to occur. It is that 
capability which makes educators eager to harness the potential of 
multimedia teclmologles. Despite that, educators and decision-makers 
need to see some evidence regarding the instructional effectiveness of 
multimedia in advance if they are to be convinced for their integration 
into educational processes. Because such an integration requires a 
considerable amount of investment. Such evidence can be obtained 
through evaluation studies which investigate tlie concept from different 
perspectives in relation to real-life educational settings. To this end, 
educators have been conducting evaluation studies since the I950's. The 
problem is that such studies have been criticised, since then, for being 
useless for a number of reasons (Clark, 1983, 1984, 1985; Salomon & 
Clark, 1977; Clark & Salomon, 1986). Similariy, multimedia simulations, 
in practice, may not live up to expectations. Thus, The main purposes of 
this paper are: 

1- to identify the main criticisms of media research, 

2- to discuss the state of current IV simulation research in the light of 
the criticisms made, 

3- and to identify the most Important points which need to be considered 
by media researchers in order to avoid making similar mistakes in the 
future. 



2. THE MAIN CRITICISMS OF IMEDIA EVALUATION STUDIES 

Let us first identify the main criticisms which have been directed towards 
media evaluation studies between the 1950*s and early 1970's, and 
between the mid 1970's and the present. That may enable us to see how 
far we have progressed, since the fifties, in terms of the quality of results 
obtained, regarding the instructional effectiveness of media. 

Just to refresh our memory about the kind of research that was carried 
out in these two periods, it might be better to examine briefly not only 
the criticisms but also the research format, the research characteristics, 
and the recommendations that had been made in order to overcome the 
^ criticisms. 
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2,11 Media Rvaluatlon Research Between IQSO^s and earlv 1970^s 

2.1.1. Research format 

- comparative (single medium vs traditional instruction) 

2.1.2. Research characteristics 

- focused on distinguishing the effects of one treatment variable to one 
outcome variable, with no recognition of individual characteristics 
(Lumsdaine. 1963; Stickwell. 1963; Copeland. 1986; Moore, et al.. 
1986). 

2.1.3. The Most Common Result 

' no significant differences between control and experimental groups 
(Stickwell. 1963; Chu & Schramm. 1967; Jamison et al,. 1974)- 
criticisms: 

- producing 'uninterpretable' findings (Seatler 1968). 

- producing 'meaningless* data (Fleming 1970). 

- not producing enough Information to determine the unique features 
of various media and whether they were effective in teaching different 
skills in various situations (Allen 1971) 

- being fruitless' (Levie & Dickie. 1973). 

Other limitations of the comparative approach have been identified as: 

- inadequate treatment definition. 

- Incomplete specification of treatment dimensions. 

- failure to measure treatment Implementation, and 

- Insufficient relevance to practical decision-making (Reeves, 1986). 

2.1.4. Recommendations made in that period 

- within-media variables "e.g. compare the effectiveness of films which 
have optical effects with those which do not" should be investigated 
(Fleming. 1970). 

- different design strategies (which relate to media characteristics, 
subject matter, and task characteristics) should be used (Campeau. 
1974). 

- more emphasis should be placed upon what is to be leamed and the 
role of the learner rather than upon the nature of the media (Levle & 
Dickie. 1973). 

- multivariate designs (in which results would not only show main 
effects but also interaction among variables which could Improve 
learning) should be used (Campeau, 1974). 

- more effort should be devoted to replication (to confirm findings of a 
particular study) and follow-up studies (to determine under what 
conditions, for which students and for which learning tasks particular 
instructional media will produce the most learning) of previous 
research (Campeau. 1974). 

- more research should be conducted on media 'content' (i.e. looking 
not only at a particular mediimi but also at very specific 
characteristics of a particular medium). 

As can be seen, all the recommendations were basically about the 
replacement of the search for the generally effective medium by a more 
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atomistic analysis of the characteristics of the medium employed in 
relation to the task demands and learners' characteristics • 



2.2) Media Evaluatton Research Between the mid 1970^^?. and the 
present 

The main development in that period was the emergence of the Aptitude- 
Treatment Interaction (ATI) approach which was initiated by Cronbach 
and Snow (1977) during the late seventies. Its main assumption was that 
learning involves an interaction of task, learner, and media 
characteristics. Thus, researchers should focus on determining the 
relationships between these factc-s and their efifect on learning rather 
than merely comparing the effectiveness of two media, did researchers 
abandon the comparative format and foUow the All approach? Although 
researchers who designed CBI studies were urged to do so (e.g. Clam, 
1985)t the literature reveals that evaluators continued to conduct large 
numbers of studies using the comparative format, hardly focused on 
ATI. and progressed very little from the 1950's and 1960's. Indeed, one of 
the major sources of support for the use of computers for instruction has 
been trie results of comparative research studies (Clark, 1985) which 
claimed increased efficiency for the new technology. The literature also 
reveals that not only computer studies but also most of the IV studies, to 
the date, have been conducted in a comparative format (Reeves, 1986; 
Cushall et al.. 1987; Slee, 1989). It could therefore be argued tliat the 
same comparative approach is still being applied for media research and 
that the findings probably deserve the same criticisms. 

It should also be pointed out that studies which followed the ATI 
approach have been criticised as well. TTius, here, we would like to briefly 
examine the criticisms in the same way. 

Aptitude-T reatment Interaction fatil approach 

2.2 A. Resectrch chcu'€icteristi€:s 

- reseai'ch data are based on comparison between control experimental 
groups 

- experiments are conducted in laboratory settings in order to 
overcome problems of precision and control (Shapiro, 1975). 

Indeed, the majority of the work on ATI have been completed in 
laboratory settings, and few attempts had been made to apply ATI to 
actual classroom settings (shapiro, 1975). 

2.2.2. The most common result 

- no significant difference between control and experimental groups 
(e.g. Cohen et al., 1981). 

2.2.3. Criticisms made 

- lack of usefiil results, because of the design employed (Reeves 1986). 

- lack of research carried out in the real life educational world and 
conducted with real materials (Salomon & Clark, 1977). 

- lack of integration in the design between media attributes and 
learner aptitudes (Clark and Angert, 1980); 
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- lack of control on variables such as instructional method, curriculum 
content, and novelty (Clark, 1983, 1984, 1985); 

- lack of guidance to the most experiments by a theoretical framework 
(Bates, 81); and 

- lack of consideration concerning individual differences in responses 
to different media (Bates, 81). 

2.2 A. Recommendations made 

' research studies employing experimental and quasi-experimental 
designs to compare instructional technologies should be abandoned 
(Reaves, 1986). 

- media research should focus on learning factors affected by the new 
medium, by the characteristics of the learners, and in the manner by 
which these media exhibited their effects (Salomon & Gardner, 1986). 

- media research should be carried out in real educational settings 
with real materials (Salomon & Clark, 1977). 

As can be seen from botJi the criticisms and the recommendations made 
for that period, problems of control over variables and rele^^ancy to the 
real world have caused the main dilemmas for the laboratory-controlled 
experimental method. While this method allows a better control over 
variables which leads to a better internal validity, and allows better 
conceptualization and imderstanding, it lacks representativeness, and 
hence, has only remote relevance to educational practice (Salomon & 
Clark, 1977). On the other hand, studies carried out in the real world of 
education, dealing with complex variables, are most often highly specific 
and do not warrant generalization. They also have a poor intemal validity 
because of the complexity of the phenomena th€y deal Avith. SaJomon & 
Clark (1977) conclude that intemal validity may be sacrificed if the study 
is a summative evaluation, which involves Judgement of worthwhileness'. 
Hie sacrifice is worth making because it allows media being studied to be 
used to its fullest advantage or to be exploited ftiUy, 

Having had a short chronological analysis of media evaluation studies 
since the fifties, let us identify what, at the present, we know about the 
instructional effectiveness of IV simulations. 



3. THE EVIDE NrR AVAT^^ ABLE ON INSTRUCTIONAL 
EFFECTIVEN ESS OF IV SIMULATIONS 

The main findings, based on an extensive literature review, can be 
identified as follows: 

- IV simulations have a positive attitudinal effect on learners, 

- effective use of IV simulations in secondarv school science lessons 
may improve standards of laboratory work and save time in 
comparison with setting up normal experiments (Doulton, 1984), 

- tutorial IV simulations in science can signiflcantiy Increase student 
performance on laboratory reports and test scores (Smith et al., 1986), 

- there is no significant difference between the cognitive effect of IV 
laboratory simulations and standard laboratory instruction in teaching 
college level pure science (Stewens, 1984), 
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- IV's visual images can help college students to understand some 
physics principles better (Cordes, 1988), 

- IV simulations can enable students to visit a real place and move 
around in time and space to investigate, sample, analyze, and test 
possible ideas within the environment (McCormick, 1987), 



4. CONCLUSION AND RECOMB«ENDATION8 FOR FUTURE 

RESEARCHERS 

In the Ught of the literature and the reported research findings on the 
instructional effectiveness of IV simulations, the following conclusions 
can be drawn: 

1- most of the studies have applied the comparative format, 

2- none of the studies has compared the cognitive and attitudinal 
effectiveness of various types of IV simulations (e.g. Surrogate, 
Competitive, Experimental) in teaching a particular learning task (e.g. 
problem solving, fact acquisition, concept acquisition, extension of 
existing kno^^^edge, transfer of knowledge and applications, 
understanding of proced\ires and processes, handling of evidence), in 
relation to learners' characteristics (e.g. gender, academic achievement, 
social backgroimd), 

3- none of the studies has compared the cognitive ?jid attitudinal 
effectiveness of various types of IV simulations in teaching different 
learning tasks, in relation to learners' characteristics, 

4- none of the studies has investigated the cognitive and attitudinal 
effectiveness of studying IV simulations in different ways (e.g. pair, 
single) in relation to (fifferent learning tasks and difierent learners 
characteristics, 

5- little work has been done on the theoretical basis of the design of IV 
simulations. 

As can be seen, there are still criticisms to be made about the way that 
new technology is evaluated. The striking point seems to be that 
although the comparative approach has been strongly criticised since the 
1950's, evaluators have continued to stick to it. TTie main reason for the 
continuity of the comparative approach seems to be that such studies 
are driven by decision makers' questions, in education, commercial, and 
military contexts where the main purpose is to prove the effectiveness of 
the medium introduced by comparing it with the existing one. The 
question that arises then is: are educators going to continue to evaluate 
multimedia applications in the same way or are they going to take the 
criticisms into account? It seems that educators should abandon the 
task of trying to prove the effectiveness of multimedia simulations in 
comparison to other technologies and concentrate on exploring their 
potential in relation to learning tasks, and leamers' characteristics. 
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